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FOREWORD 


Under  Project  7719,  Task  771904,  Headquarters  USAF  has  established  a 
requirement  with  Personnel  Research  Laboratory  for  research  on  problems  as¬ 
sociated  with  officer  performance  evaluation  and  promotion  selection.  This  report 
is  the  first  of  a  series  presenting  the  results  of  analyses  of  officer  promotion 
actions  and  describing  experiments  with  a  view  to  developing  procedures  which 
will  increase  reliability.  In  this  report,  the  reliability  of  officer  promotions  as' 
presently  constituted  is  assessed. 

Special  acknowledgment  is  giver,  to  the  Promotion  Board  Secretariat, 
Headquarters  USAF,  for  their  assistance,  cooperation,  and  guidance  in  the  design 
of  the  studies  and  the  data  collection,  and  their  wholehearted  support  of  the 
research  effort.  All  distribution  of  this  report  is  controlled  by  the  Secretariat 
because  the  information  relates  to  Ait  Force  management  policies. 

This  technical  report  has  been  reviewed  and  is  approved. 


John  Patterson,  Col  USAF 
Commander 


A.  Carp 
Technical  Director 
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ABSTRACT 

Data  from  actions  of  the  FY  1962  Promotion  Boards  for  majors,  lieutenant 
colonels,  and  colonels  were  used  to  estimate  reliability  of  decisions  and  the 
relationship  of  Promotion  Scores  to  Officer  Effectiveness  Reports  (OERs).  A 
series  of  statistical  analyses  showed  that:  (1)  the  evaluations  and  resulting 
recommendations  regarding  promotion  ate  reliable;  (2)  from  80  to  90  percent  of 
the  decisions  would  have  been  concurred  in  by  a  hypothetical  second  board; 

(3)  the  nearer  an  eligible  is  placed  tu  the  selection  cutoff  score,  the  greater  the 
likelihood  that  the  promotion  decision  about  him  would  have  been  reversed  by  a 
second  board;  and  (4)  while  mean  OER  is  related  to  Promotion  Score,  it  has  been 
shown  that  other  factors  also  contribute  to  this  score.  An  appendix  describes  the 
method  of  estimating  reliability  of  panel  scores  and  board  decisions. 
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AN  ANALYSIS  OF  OFFICER  PROMOTION  ACTIONS 


I.  INTRODUCTION 

In  August  1962  a  series  of  studies  of  officer  promotion  actions  were  initiated  to  analyze 
officer  promotion  actions  from  the  standpoint  of  reliability  and  stability;  to  devise  and  carry 
Out  analyses  and  experimental  studies  leading  to  possible  increases  in  the  effectiveness  of 
the  officer  ptomotion  system;  and  to  propose  those  changes  which  the  analyses  indicate  would 
feasibly  result  in  increased  efficiency.  A  series  of  reports  of  these  studi'.*  is  planned,  of 
which  the  present  report,  is  the  first.  Reports  immediately  forthcoming  will  discuss  the  feasi¬ 
bility  of  using  electronic  computers  to  assist  Promotion  Boards  and  various  ways  in  which  the 
reliability  of  promotion  board  evaluations  and  officer  promotion  actions  may  be  increased. 
Later  reports  will  present  the  results  of  studies  in  which  factors  such  as  number  of  panel 
members,  amount  and  type  of  material  made  available  to  the  panel  members,  and  the  method  of 
presentation  of  this  material  are  systematically  varied.  Finally,  a  summary  report  will  bring 
together  the  results  cf  the  studies  and  analyses  and  suggest  certain  specific  changes  in  the 
present  system  which  might  be  tried  out  by  one  or  more  actual  Promotion  Boards. 

This  first  report  describes  analyses  which  were  undertaken  to  assess  the  consistency 
with  which  officer  promotion  decisions  are  made.  Attention  is  focused  on  both  the  extent  to 
which  two  or  more  Promotion  Board  panels  would  have  made  die  same  decision,  and  on  the 
consistency  with  which  certain  available  information  about  the  promotion-eligible  officer  is 
used  in  arriving  at  a  decision  about  him.  Insights  gained  from  analyses  of  this  type  may  lead 
to  the  formulation  of  board  procedures  designed  to  enhance  the  consistency  with  which  promo¬ 
tion  decisions  are  made. 


it.  PROMOTION  BOARD  PROCEDURE 

Officer  promotion  recommendations  are  made  by  special  boards  which  are  convened  at 
Headquarters  USAF.  A  detailed  description  of  the  procedures  followed  by  Promotion  Boards 
is  not  required  here,  but  a  general  description  is  given.  An  officer  Promotion  Board  is  composed 
of  senior  Air  Force  officers  drawn  from  the  Air  Force  at  large  to  serve  on  the  Promotion  Board. 

In  advance  of  the  date  on  which  the  Board  convenes,  records  of  officers  to  be  considered  for 
promotion  are  assembled  at  a  central  point  and  made  ready  for  the  Board’s  use. 

The  Board  consists  of  a  president,  recorders,  and  a  number  of  three-member  promotion 
panels.  The  Board  is  briefed  on  its  mission,  Air  Force  policy  with  regard  to  officer  ,  romotions 
to  the  grade  for  which  the  Board  is  to  make  selections,  and  Air-Force-wide  trends  in  officer 
ratings  which  may  have  bearing  on  the  deliberations  of  the  Board. 

Following  the  briefing,  all  Board  members  evaluate  a  sample  of  the  records  as  a  "Trial 
Run”  for  training  and  orientation  purposes.  "Trial  Run”  ratings,  both  for  individual  Board 
members  and  for  panels,  are  posted  and  discussed  with  the  entire  Board.  The  purpose  of  the 
Trial  Run  is  to  assist  Board  members  in  establishing  a  "standard”  against  which  they  will 
evaluate  officers  being  considered  for  promotion.  For  this  reason,  the  records  evaluated 
during  the  "Trial  Run”  have  been  carefully  preselected  to  cover  the  range  of  officer  perform¬ 
ance.  This  allows  each  member  of  the  Board  to  familiarize  himself  with  the  range  of  perform¬ 
ance  and  background  he  may  encounter  in  evaluating  officer  records  and  to  set  his  standards 
for  various  evaluations  in  light  o'  'his  infoimation. 
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Once  the  Trial  Run  has  been  completed,  care  is  exercised  to  see  that  ot'ficer  records  are 
distributed  to  the  Board’s  panels  in  a  random  manner.  A  record  is  rated  by  all  three  memberc 
of  the  panel  to  which  it  is  assigned,  and  a  Promotion  Score  is  obtained  by  summing  the  three 
individual  ratings.  Certain  procedures  are  employed  to  obtain  additional  evaluation  of  records 
on  which  there  is  evidence  of  considerable  disagreement  among  panel  members.  Final  Board 
selections  are  made  within  panels.  Each  panel  selects  its  proportionate  share  of  the  prcmot'on 
quota  as  a  safeguard  against  promotion  of  officers  who  were  evaluated  by  an  "easy"  panel 
while  equally  deserving  officers  evaluated  by  a  "hard"  panel  are  passed  over. 


III.  CONCEPTS  OF  RATER  RELIABILITY 

Thus  it  is  readily  seen  that  consistency  of  evaluations  becomes  crucial  in  evaluating  the 
appropriateness  of  a  given  Board's  promotion  decisions.  Several  different  sorts  of  consistency 
might  be  considered  in  analyzing  the  evaluations  made  by  a  Promotion  Board.  At  one  level, 
one  mi^t  be  concerned  with  the  consistency  with  which  a  given  panel  makes  its  evaluations - 
have  the  "standards"  against  which  the  panel’s  members  make  their  evaluations  remained  con¬ 
stant.’  If  the  same  record  were  evaluated  by  the  panel  at  two  different  points  in  time,  would  it 
receive  the  same  evaluation  at  both  times? 

At  another  level,  one  might  be  concerned  with  the  extent  to  which  different  promotion 
panels  would  rank-order  eligible  officers  in  the  same  way.  Different  panels  might  be  ordering 
eligible  officers  in  the  same  way,  but  applying  different  score  values  to  them.  In  this,  case 
overall  promotion  actions  agree,  but  it  is  probably  easier  to  understand  and  controi  a  board 
where  the  panels  are  producing  similar  score  distributions. 

Since,  in  the  practical  situation,  one  rarely  has  data  that  would  allow  for  a  direct  com¬ 
parison  of  ratings  across  panels,  one  might  ask  about  the  extent  to  which  different  panels  use 
infoimation  about  promotion-eligible  officers  in  a  consistent  manner  in  evaluating  them.  Are 
those  things  judged  "important"  by  one  panel  given  equal  importance  in  the  evaluations  tend¬ 
ered  by  other  panels? 

Vanasek',  in  studying  the  relationships  between  ratings  rendered  by  different  1957 
officer  augmentation  panels,  found  that  the  statistical  reliabilities  were  high  and  that  judg¬ 
ments  did  not  differ  significantly  between  panels.  He  indicated,  however,  that  since  individual 
lives  were  closely  bound  up  with  each  selection  resulting  from  panel  judgments  it  was  necessary 
to  investigate  the  adequacy  of  different  panel  judgments  in  terms  of  final  selections.  In  effect, 
he  was  pointing  out  that  small  rating  differences  near  a  cutoff  score  might  be  statistically 
nonsignificant  but  still  result  in  a  number  of  selection  differences  from  one  panel  to  another. 

He  found  that,  for  a  group  of  officers  with  5  years  of  service,  from  14  to  23  percent  of  the 
selections  differed  depending  upon  the  panel  judging  the  folders.  In  the  case  of  a  group  of 
officers  with  10  years  of  service,  this  percentage  ranged  from  7  to  11. 

A  series  of  studies  in  1963,  examined  the  reliability  (or  consistency)  of  officer  Promotion 
Board  actions,  and  the  extent  and  consistency  with  which  the  Mean  Officer  Effectiveness 
Rating  (Mean  OER)  is  used  in  arriving  at  these  decisions.  Three  of  these  are  described  in  the 
next  sections. 


'  Unpublished  manuscript,  "Prediction  Selection”  by  F.  J.  Vanasek,  1958. 
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IV.  RELATIONSHIPS  BETWEEN  N(»1INATINC  AND  PROMOTION  SCORES 


When  two  or  more  raters  or  ratingpanels  operate  with  the  same  frame  of  reference,  one 
would  expect  a  high  relationship  amot.g  the  ratings  they  assign  to  a  given  group  of  officers. 

The  closer  raters  are  to  a  common  frame  of  reference  with  regard  to  the  characteristic  being 
rated,  the  higher  will  be  agreement  among  the  ratings  they  assign.  This  "interrater"  agree.men- 
is  different  from  "intrarater**  agreement,  or  intemal  consistency.  The  latter  may  be  thought  of 
as  an  index  of  the  extent  to  which  a  rater  consistently  applies  his  frame  ol  referc.'tce,  and  elimi¬ 
nates  "chance**  variations  from  the  ratings  he  assigns. 

More  often  than  not,  the  data  available  from  activities  of  personnel  boards  are  such  that 
inter-rater  agreement  must  be  estimated  statistically;  rarely  are  data  available  which  allow  for 
direct  comparison  of  the  decisions  made  by  different  panels  or  boards.  The  only  group  of  offi¬ 
cers  for  whom  data  of  this  sort  are  available  are  those  considered  for  promotion  to  temporary 
colonel  each  year.  Records  of  eligible  lieutenant  colonels  are  reviewed  each  year  by  a  Nomi¬ 
nating  Board  which  then  selects  a  smaller  number  of  the  eligibles  for  consideration  by  the 
Temporary  Colonel  Promotion  Board.  For  officers  who  were  nominated  by  the  FY  62  Colonel 
Nominating  Board,  Nominating  Board  ratings  and  Promotion  Board  ratings  were  compared  to 
gain  some  idea  of  the  extent  to  which  these  different  boards  agree. 

The  FY  62  Nominating,Board  considered  records  of  2850  lieutenant  colonels,  and  nomi¬ 
nated  1200  of  them  to  the  Promotion  Board,  The  Nominating  Board  was  composed  of  7  three- 
member  rating  panels,  and  the  Promotion  Board  was  composed  of  4  rating  panels. 

The  1200  lieutenant  colonels  nominated  to  the  Promotion  Board  were  divided  into  28  sub¬ 
groups  on  the  basis  of  the  combination  of  Nominating  Board  and  Promotion  Board  rating  panels 
rating  them  (i.e.,  officers  rated  by  Nominating  Board  panel  1  and  by  promotion  panel  1  formed 
one  subgroup,  etc.).  Relationships  between  Nominating  Scores  and  Promotion  Scores  were 
established  for  each  of  these  subsaroples.* 

There  is  considerable  variation  (from  subgroup  to  subgroup)  in  the  strength  of  this  re¬ 
lationship.  This  variation  may  be  accounted  for  in  part  by  the  fact  that  the  Promotion  Board 
does  not  see  the  records  of  the  lowest  scoring  individuals,  but  this  can  by  no  means  entirely 
explain  the  differences.  For  example,  from  Table  XV  it  can  be  seen  that  those  officers 
selected  by  nominating  panel  1  and  later  rated  by  promotion  panel  3,  and  those  officers 
selected  by  nominating  panel  7  and  later  rated  by  promotion  panel  3  have  almost  identical 
dispersion  in  their  ratings;  none-the-less,  ratings  of  those  officers  selected  by  nominating 
panel  7  are  much  more  highly  related  to  promotion  panel  3  ratings  than  are  nominating  panel  1 
ratings. 

For  each  of  the  1200  officers,  the  promotion  decision  that  the  Nominating  Board  would 
have  made  if  it  had  served  as  the  Promotion  Board  was  determined.  These  decisions  were 
compared  with  those  thar  were  actually  made  by  the  Promotion  Board.  For  each  nominating 
panel  separately.  Table  1  shows  the  extent  of  agreement  between  Promotion  Board  decisions 
and  those  that  would  have  been  made  by  that  nominating  panel.  One  cannot  say  with  certainty 
what  decision  the  Promotion  Board  would  have  made  about  those  officers  not  nominated  to  it 
by  the  Nominating  Board.  Table  I  suggests  that  most  of  them  would  not  have  been  recommended 
for  promotion;  percentage  of  agreement  between  the  Boards  is  relatively  high,  and  the  further  an 
officer*s  rating  is  below  the  cutoff  score,  the  less  likely  it  is  that  a  promotion  decision  about 
him  would  be  reversed  by  a  second  board. 


^  These  relationships  are  presented  in  Table  15,  Appendix  I,  for  readers  who  wish  to  see  the 
I  ortelation  data. 
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Table  1.  AgrPemenl  Between  NominatingBoard  and  Bromotion 
Board  Decisions,  FY  62  Temporary  Colonels 


NOMINATING 

PANEL 

DECISION* 

PROMOTION  BOARD 
DECISION* 

AGREEMENTS 

0 

1 

NO. 

percent'" 

1 

1 

27 

43 

118 

70.66 

0 

75 

22 

2 

1 

12 

61 

146 

83.43 

0 

85 

17 

3 

1 

20 

50 

132 

79.04 

0 

82 

15 

4 

1 

23 

40 

109 

72.18 

0 

69 

19 

5 

1 

14 

59 

140 

80.00 

0 

81 

21 

6 

1 

31 

53 

149 

74.50 

0 

96 

20 

7 

1 

9 

62 

138 

81.66 

0 

76 

22 

*i  •:  promote;  0  =  don't  promote. 

’’Percent  of  cases  rated  by  the  Promotton  Board. 


These  data,  along  with  data  on 
relationships  between  the  numerical 
ratings,  suggest  that,  while  there  is 
considerable  agreement  between  Boards 
with  regard  to  promotion  decisions,  there 
is  considerable  disagreement  between 
some  specific  panels  with  regard  to  the 
numerical  ratings.  This  in  turn  suggests 
that  the  "frame  of  reference’’  of  some 
panels  differs  considerably  from  that  of 
other  panels.  These  disagreements  will 
result  in  differences  in  promotion  deci¬ 
sions  near  a  "cutting  point’’;  the  panels 
will  tend  to  agree  about  promotion  o: 
nonpromotion  of  "high”  and  "low”  rases 
with  greater  frequency  than  for  "intei- 
mediate”  cases. 


V.  RELIABILITY  ESTIMATES  OF  PROMOTION 
SCORES 

It  is  possible  to  estimate  statistically 
the  extent  to  which  a  second  group  of  raters 
would  agree  with  an  observed  setof  ratings. 
The  statistical  procedures  involved  assume 


Table  II.  Board-to-Boaid  Reliability, 
FY  62  Temporary  Majors 


BOARD  1  SCORES 


'S 

14 

17 

IS 

_19 

20 

21 

22 

23 

24 

25 

26 

27 

28 

29 

30 

TOTAL 

30 

3 

6 

18 

12 

39 

29 

6 

23 

23 

32 

lu 

94 

28 

1 

7 

43 

95 

50 

36 

6 

238 

B 

42 

27 

1 

18 

47 

161 

140 

50 

21 

2 

440 

ts 

Cl 

26 

1 

10 

no 

171 

269 

227 

24 

5 

817 

25 

1 

8 

59 

329 

322 

222 

72 

6 

1 

1020 

•V 

& 

*• 

u 

24 

1 

7 

40 

159 

498 

232 

105 

17 

1 

1060 

23 

6 

35 

107 

228 

359 

'j7 

19 

2 

843 

22 

4 

28 

81 

124 

149 

124 

16 

2 

528 

21 

3 

12 

58 

93 

75 

48 

21 

1 

311 

V 

20 

2 

10 

22 

60 

53 

28 

8 

2 

185 

t 

19 

2 

7 

18 

27 

32 

12 

3 

101 

1 

18 

4 

12 

16 

11 

8 

2 

53 

17 

7 

10 

7 

3 

1 

28 

16 

5 

4 

1 

10 

15 

2 

2 

Total  0 

20 

35 

55 

79 

194 

284 

386 

662  1462 

883 

827 

579 

160 

113 

30 

5769 

r..  =  .88 
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that  'he  second  group  of  raters  arc  exactly  like  the  group  from  whom  observed  ratings  arc 
obtained,  and  that  they  render  ratings  from  the  same  frame  of  reference.  If  these  assumptions 
ate  not  met,  an  over  estimate  of  the  extent  of  agreement  is  obtained.  In  essence,  Aese  esti¬ 
mates  amount  to  a  statistical  detctmination  of  the  extent  to  which  the  judgments  jf  the  raters 
in  the  observed  group  vary  together.*  It  should  be  remembered  that  while  any  or  all  of  the 
raters  in  the  observed  group  may  be  rating  with  perfect  internal  consistency,  the  in^x  of 
agreement  reflects  only  the  extent  to  which  their  ratings  agree  with  other  raters  in  die  group. 

The  estimation  technique  assumes  that  variation  in  ratings  which  is  not  related  to  ratings 
assigned  by  other  raters  is  "chance"  or  random  variation. 

Evaluation  scores  from  three  FY  62  Promotion  Boards  (Temporary  Major.  Temporal 
Lieutenant  Colonel,  and  Temporary  Colonel)  have  been  analyzed  and  their  reliabilities  (i.e., 
"agreement”)  estimated. 

The  reliabilities  were  estimated  by  determining  the  variance  of  ratings  assigned  by  each 
of  the  three  panel  members  and  comparing  these  values  with  the  variance  of  the  evaluation 
scores  based  upon  the  sum  of  the  three  ratings.  It  has  been  demonstrated  statistically  (1. 
p.  223)  that  this  procedure  yields  fairly  accurate  reliability  estimates. 

In  Table  D  re'iability  data  for  the  FY  62  Temporary  Major  Promotion  Board  are  presented; 
similar  data  for  the  FY  62  Temporary  Lieutenant  Colonel  and  Temporary  Colonel  Promotion 
Boards  appear  in  Tables  3  and  4.  The  manner  in  which  these  estimates  were  compute  ,  a^ong 
with  panel-by-panel  reliability  estimates  ate  repotted  in  Appendix  U  for  the  benefit  of  tea  ers 
interested  in  mote  detailed  information.  The  data  in  each  table  ate  presented  in  the  « 
a  two-way  distribution  of  evaluation  scores.  The  horizontal  axis  in  each  table  represents  the 
distribution  of  evaluation  scores  actually  assigned  the  group  of  eligible  officers  l>y  the  Promo¬ 
tion  Board.  The  vertical  axis  represents  the  distribution  of  scores  expected  if  anoAer  board 
were  to  independently  evaluate  Ae  same  records.  The  values  at  the  bottom  of  each  column 
indicate  Ae  number  of  officers  receiving  Ae  Board  1  Promotion  Score  shown  at  the  top  of  Aat 
column.  The  figures  in  each  column  indicate  the  distribution  of  Promotion  Scores  expected 
from  a  second  board.  For  example,  looking  at  Table  II  (FY  62  Temporary  Ma,ors)  it  can  be 
seen  Aat  Ae  P.-amotion  Board  assigned  an  evaluation  score  of  27  to  579  eligibles.  Were  a 
second  board  to  re-evaluate  Ae  records  of  these  579  eligibles,  3  of  them  would  be  gi..n  a 
Promotion  Score  of  30,  23  a  score  of  29.  etc.  The  other  columns  and  the  oAer  tables  can  be 

interpreted  in  the  same  manner. 

It  can  be  seen  from  these  tables  that  fairly  high  agreement  between  boards  would  be 
expected  wiA  regard  to  Promotion  Scores  assigned.  Since  Ae  score  assigned  to  a  «««!  is 
not  so  imponant  as  the  meaning  of  that  score  wiA  regard  to  final  promotion  decision,  the  data 
contained  in  Tables  II,  lU,  and  IV  have  been  recast  in  terms  of  expected  interboard  agreement 
with  regard  to  promotion  and  presented  as  Tables  V,  VI,  and  VII. 

In  so  reducing  the  data,  a  Board  1  "cutoff”  score  which  yields  approximately  the  per- 
centage  of  promotions  made  by  Ae  board  was  selected.  A  cut  off  for  Board  2  (the  hypothetica 
second  board)  was  established  by  finding  that  point  in  the  Beard  2  distraction  which  would 
yield  approximately  the  same  promotion  ratio  as  that  used  for  Board  1.  Where  necessary  to 
obtain  Ae  same  promotion  ratio  for  Board  2,  interpolations  into  a  score  interval  were  made. 

It  may  be  seen  from  Table.v  V,  VI,  and  VII  Aat  two  separate  boards  would  be 
to  agree  on  promotion  decisions  lor  84.7  percent  of  the  officers  considered  for  maj^ar,  or  91.6 
percent  of  the  officers  considered  for  lieutenant  colonel,  and  for  79.1  percent  of  t^  of  leers 
Lnsidered  for  colonel.  (TTiese  percentages  are  obtained  from  the  of  Ae  number  of  officers 
"Above  Cutoff”  on  both  boards  and  Ae  number  "Below  Cutoff"  on  both  boards.) 


’Sec  Appendix  II  for  an  explanation  of  ihe  method. 

5 


Table  III.  Buard-to-Board  Reliability, 
FY  62  Temporary  Ll  Colonels 


BOARD  1  SCORES 

TO'AL 

IS 

16 

17 

18 

19 

20 

2) 

22 

23 

24 

25 

26 

27 

28 

29 

30 

30 

7 

16 

14 

20 

58 

29 

1 

8 

38 

49 

24 

13 

133 

28 

2 

12 

48 

128 

87 

23 

7 

307 

£  27 

1 

23 

71 

148 

218 

83 

12 

2 

558 

g  26 

2 

12 

116 

206 

240 

197 

41 

4 

818 

t«  25 

2 

13 

56 

321 

305 

197 

87 

11 

992 

^  24 

1 

2 

15 

58 

142 

428 

234 

84 

22 

986 

§  23 

4 

11 

64 

138 

190 

322 

95 

22 

3 

349 

ffl  22 

1 

18 

45 

139 

154 

128 

121 

19 

1 

626 

S  21 

1 

8 

48 

95 

160 

118 

43 

23 

2 

498 

S  20 

1 

5 

29 

76 

100 

96 

35 

8 

2 

352 

t  19 

4 

16 

52 

76 

54 

28 

6 

1 

237 

^18 

2 

12 

29 

47 

48 

19 

6 

1 

164 

*  17 

4 

20 

25 

24 

18 

3 

1 

95 

16 

7 

17 

12 

6 

4 

46 

15 

8 

9 

3 

1 

1 

22 

Total 

21 

63 

91 

168 

294 

329 

511 

525 

581 

1358 

945 

749 

700 

287 

77 

41 

67*f  1 

=  .90 

Table  IV.  Board-to-Board  Reliability, 

FY  62  Temporary  Colonels 

BOARD  1  SCORES 

TOTAL 

15 

14  17 

18 

19 

20 

21 

22 

23 

24 

25 

26 

27 

28 

29 

30 

30 

1 

1 

2 

29 

2 

6 

5 

1 

1 

15 

28 

I 

5 

10 

15 

9 

2 

1 

43 

27 

1 

5 

16 

28 

30 

11 

1 

92 

« 

if 

26 

2 

7 

21 

41 

46 

32 

8 

1 

158 

35 

25 

2 

8 

24 

46 

57 

43 

21 

4 

205 

(A 

24 

1 

6 

17 

44 

60 

50 

25 

7 

1 

211 

•s 

23 

1 

4 

15 

44 

51 

52 

26 

8 

2 

203 

1 

22 

2 

8 

23 

42 

35 

14 

7 

2 

133 

21 

1 

3 

11 

21 

18 

14 

5 

1 

74 

20 

1 

5 

8 

11 

8 

3 

1 

37 

«> 

19 

1 

3 

4 

4 

2 

14 

g. 

18 

1 

1 

1 

3 

s 

17 

0 

16 

0 

15 

0 

Total 

0 

0  0 

3 

15 

37 

83 

141 

179 

205 

203 

164 

113 

39 

6 

2 

1190 

=  .77 
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Table  y.  Board-to*Board  Srlection  Comparison, 


FY  62  Temporary  Majors 

BOARD  1 

BELOW  CUTOFF 

ABOVE  CUTOFF 

TOTAL 

BOARD  2 

N 

PERCENT 

N 

PERCENT 

N 

PERCENT 

Above  cutoff 

442 

7.7 

3614 

62.6 

4056 

70.3 

Below  cutoff 

1273 

22.1 

440 

7.6 

1713 

29.7 

Total 

1715 

29.8 

4054 

70.2 

5769 

100.0 

Table  VI.  Board-to-Board  Selection  Comparison, 
FY  62  Temporary  Lt  Colonels 


BOARD  1 


BOARD  2 

BELOW  CUTOFF 

ABOVE  CUTOFF 

TOTAL 

N 

PERCENT 

N 

PERCEN  r 

N 

PERCENT 

Above  cutoff 

281 

4.2 

4983 

73.9 

5264 

78.1 

Below  cutoff 

1196 

17.7 

281 

4.2 

1477 

21.9 

Total 

1477 

21.9 

5264 

78.1 

6741 

100.0 

Table  Vll.  Board-to-Board  Selection  Comparison, 
FY  62  Temporary  Colonels 

BOARD  1 

BELOW  CUTOFF 

ABOVE  CUTOFF 

TOTAL 

BOARD  2 

N  PERCENT 

N  PERCENT 

N 

PERCENT 

Above  cutoff 

124  10.4 

402  33.8 

526 

44.2 

Below  cutoff 

5.39  45.3 

125  10.5 

664 

55.8 

Total 

663  55.7 

527  44.3 

1190 

100.0 

Some  idea  of  the  cases  on  which  separate  boards  would  be  most  likely  to  disagree  can 
be  obtained  by  expanding  the  data  in  Tables  V,  VI,  and  Vll  such  that  a  distribution  of  Promo¬ 
tion  Scores  from  the  actual  board  (Board  I)  against  expected  promotion  selections  by  the 
hypothetical  second  board  (Board  2)  may  be  inspected.  These  data  are  shown  in  Tables 
VIII,  IX,  and  X. 

It  can  be  seen  from  these  tables  that  the  nearer  the  cutoff  score  a  ratee’s  Promotion 
Score  places  him,  the  less  reliable  (or  consistent)  is  the  promotion  decision  about  him;  and 
the  more  likely  it  is  that  a  second  board  would  have  arrived  at  the  opposite  decision  about 
him.  For  example,  chances  ate  about  1  in  2  (48.8%)  that  a  captain  who  was  passed  over  with 
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a  score  of  23  (just  below  the  cutoff)  by  the  FY  62  Majors  Board  would  have  been  selected  had 
the  records  been  considered  by  a  second  board;  these  probabilities  drop  off  as  the  score  as¬ 
signed  by  the  Board  gets  lower,  so  that  chances  are  only  about  2  in  100  that  a  captain  passei' 
over  widi  a  score  of  20  by  the  Board  would  have  been  promoted  by  a  second  board. 


VI.  RELATIONSHIP  BFTVEEN  MEAN  OER  AND  PROMOTION  SCORE 

Past  studies  (2,  3}  have  indicated  that  an  officer’s  Mean  Effectiveness  Racing  (Mean 
OER)  is  more  highly  related  to  personnel  board  decisions  about  him  than  is  any  other  available 
measure.  It  was  hypothesized  that,  in  assij^ing  promotion  ratings,  the  rater  gains  an  impres¬ 
sion  of  the  mean  OER  from  inspection  of  the  individual  OERs  for  officers  who  are  quite  high 
or  low  on  this  variable;*  but  that  a  number  of  variables  are  taken  into  consideration  for  officers 
widi  intermediate  level  Mean  OERs  in  an  effort  to  evaluate  them  "fairly.” 

Stated  differently,  one  would  expect  raters  to  feel  that  only  quickly  obvious  major  attri¬ 
butes  or  deficiencies  would  alter  final  promotion  decisions  about  officers  whose  effectiveness 
ratings  have  been  consistently  high  or  low;  on  the  other  hand,  officers  who  have  performed  at 
an  intemediate  level  may  well  have  a  promotion  decision  reversed  by  consideration  of  special 
or  unusual  accomplishments,  abilities,  or  training.  The  purpose  of  this  analysis  is  to  deter¬ 
mine  the  extent  of  relationship  between  Promotion  Scores  and  Mean  OERs,  and  to  examine  the 
nature  of  that  relationship. 

For  each  of  the  FY  62  Promotion  Boards  (Major,  Lieutenant  Colonel,  and  Colonel)  sep¬ 
arately,  and  for  the  FY  62  Colonels  Nominating  Board,  officers  considered  by  die  board  were 
divided  into  subgroups  based  on  Mean  OER  intervals.  For  each  of  these  subgroups,  the  mean 
and  standard  deviation  of  the  Promotion  Score  was  computed.  In  addition,  the  correlation 
between  Mean  OER  and  Promotion  Score  was  computed  for  each  of  the  four  boards.  These 
data  are  shown  in  Tables  XI,  XII,  XIII,  and  XIV. 

If  the  hypothesis  under  consideration  is  tenable,  one  would  expect  the  relationship  be¬ 
tween  Promotion  Score  and  Mean  OER  to  assume  a  foim  in  which  Mean  Promotion  Score  within 
successively  lower  Mean  OER  intervals  drops  off  in  a  fairly  linear  manner  (i.e.,  decreases 
fairly  uniformly);  moreover,  one  would  expect  the  Promt-ion  Score  standard  deviation  (which 
is  a  measure  of  variability)  to  be  smaller  in  die  high  and  low  Mean  OER  intervals  than  in  the 
intermediate  intervals. 

It  appears  that  the  relationship  between  Mean  OER  and  Promotion  Score  is  linear,  (For 
the  Temporary  Colonel  Promotion  Board  the  mean  Promotion  Score  is  higher  in  the  lowest  Mean 
OER  interval  than  in  the  next  higher  interval,  but  this  mean  is  based  on  only  two  cases.) 
However,  there  is  little  evidence  for  greater  Promotion  Score  difference  in  the  middle  Mean 
OER  intervals  than  at  the  extremes,  except  for  the  Colonels  Nominating  Board.  It  may  be  that 
the  Nominating  Board's  ratings  tend  to  conform  to  the  hypothesis  regarding  Mean  OER  and 
Promotion  Score  relationships  because  the  Nominating  Board  members  assume  that  the  Promo¬ 
tion  Board  will  wish  to  examine  all  high  Mean  OER  folders  but  will  not  wish  to  examine  low 
Mean  OER  folders.  In  this  sense,  the  task  assumed  by  the  Nominating  Board  might  be  regarded 
as  different  from  that  of  the  Promotion  Board. 

For  the  Lieutenant  Colonel  Promotion  Board  there  is  some  increase  in  Promotion  Score 
•  difference  from  the  highest  Mean  OER  interval  to  die  lower  intervals;  and  for  the  Major 


*The  Mean  OER  itself  is  never  available  to  Piomotion  Board  members. 
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Table  VIU.  Distribution  of  Board-to-Board  Selection  Disaitreenients,* 
FY  62  Temporary  Majors 


BOARD  1 
SCORES 

ABOVE 

CUTOFF 

BELOW 

CUTOFF 

TOTAL 

XOIS. 

AGREE. 

MENTS*’ 

%  DISTR  OF 
DISAGREE. 
MENTS' 

30 

30 

30 

29 

113 

113 

28 

160 

160 

27 

578 

1 

579 

.2 

.1 

26 

814 

13 

827 

1.6 

1.5 

25 

815 

68 

883 

7.7 

7.7 

24 

1104 

3S8 

1462 

24.5 

40.6 

23 

323 

339 

662 

48.8 

36.6 

22 

93 

293 

386 

24.1 

10.5 

21 

22 

262 

284 

7.8 

2.5 

20 

4 

190 

194 

2.1 

.5 

19 

79 

79 

18 

55 

55 

17 

35 

35 

16 

20 

20 

15 

Total 

4056 

1713 

5769 

15.3 

100.0 

“'Ditagteement’*  entries  appear  in  italics. 

**  Percent  of  cases  in  the  Board  1  score  interval  on  whom  promotion  "disagreement**  occurs. 
‘Percentage  distribution  on  Board  1  score  of  **disagteement**  cases. 


Table  IX.  Distribution  of  Board*to-Board  Selection  Disagreements,* 
PY  62  Temporary  Lt  Colonels 


BOARD  1 

ABOVE 

BELOW 

XOIS. 

AGREE. 

X  DISTR  OF 
DISAGREE. 

SCORES 

CUTOFF 

CUTOFF 

TOTAL 

MENTS** 

«ENTS‘ 

30 

42 

42 

29 

77 

77 

28 

287 

287 

27 

700 

700 

26 

749 

749 

25 

941 

4 

945 

.4 

.7 

24 

1321 

37 

1358 

2.7 

6.6 

23 

516 

65 

581 

11.2 

11.6 

22 

350 

175 

525 

33.3 

31.1 

21 

206 

305 

511 

40.3 

36.7 

20 

53 

276 

329 

16.1 

9.4 

19 

21 

273 

294 

7.1 

3.7 

18 

1 

167 

91 

168 

.6 

.2 

17 

91 

16 

63 

21 

63 

15 

21 

Total 

5264 

1477 

6741 

8.4 

100.0 

i 

i 


I 


1 


*'*Disagreement'*  entries  appear  in  ttahcs. 

’’Percent  of  cases  in  the  Board  1  score  interval  on  whom  promotion  **disagreefflent*’  occura. 
‘Percentage  distribution  on  Board  I  score  of  **disagreement**  cases. 
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Table  X.  Dialribation  of  Board-tO'Bovd  Selection  Disagreements,' 
FY  62  Temporary’  Colonels 


BOARD  1 
SCORES 

ABOVE 

CUTOFF 

BELOW 

CUTOFF 

TOTAL 

XDIS* 

agree¬ 

ments'* 

*DISTR  CF 
DISAGREE¬ 
MENTS' 

30 

2  1 

2 

29 

6 

6 

28 

38  1 

1 

39 

2.6 

.4 

27 

104  , 

9 

113 

8.0 

3.6 

26 

130  ' 

34 

164 

20.7 

13.7 

25 

122  , 

81 

203 

39.9 

32.3 

24 

76  1 

129 

205 

37.1 

30.5 

23 

35 

144 

179 

19.6 

14.1 

22 

11  ' 

130 

141 

7.8 

4.4 

21 

2  1 

81 

83 

2.4 

.8 

20 

1 

37 

37 

19 

1 

15 

15 

18 

3 

3 

17 

1 

16 

15 

1 

Total 

526  1 

664 

1190 

20.9 

100.0 

* ''Disagreement**  entries  appear  in  ilatics. 

’’Percent  of  cases  in  the  Board  1  score  interval  on  «hom  promotion  **disajreemeot**  occurs. 
'’Percentage  distribution  on  Board  1  score  of  "disagreement**  cases. 


Table  XL  Relationship  Between  Mean  OER  and  Promotion 
Score  for  Officers  Considered  for 
Promotion  to  Major 


MEAN  OER 

INTErVAL 

N 

PROMOTION  SCORE 

MEAN  SO 

8.5  -  9.0 

40 

27.70 

1.75 

7.5  -  8.4 

1304 

25.97 

1.64 

6.5  -  7.4 

2623 

24.00 

1.51 

5.5 -6.4 

1765 

22.20 

1.74 

4.5- 5.4 

347 

19.73 

1.94 

3.5 -4.4 

37 

18.05 

2.46 

2.5  -  3.4 

1 

16.00 

0.00 

r  =  .69 

r 


Promotion  Board,  Promotion  Score  dispersion  decreases  from  the  highest  Mean  OER  interval  to 
the  intermediate  intervals  and  dien  begins  to  increase  again. 

Relationships  between  Mean  OERs  and  Promotion  Scores  are  moderately  high;  correlation 
for  the  Colonels  Promotion  Board  is  considerably  less  than  for  the  other  boards.  This  lower 
correlation  can  be  explained  in  part  by  the  restriction  in  range  of  performance  produced  by  the 
Nominating  Board’s  screening  of  eligible  officers. 


Table  XU.  Relationship  Between  Mean  OER  and  Promotion 
Score  for  Officers  Considered  for  Promotion  to  Lt  Colonel 


PROMOTION  SCORE 

MEAN  OER  - 

INTERVAL  N  MEAN _ ^ 


8.5  -  9.0 

130 

26.51 

2.03 

7.5  -  8.4 

2347 

25.43 

1.93 

6.5  -  7.4 

2783 

23.50 

2.21 

5.5  -6.4 

945 

21.43 

2.37 

4.5  -  5.4 

136 

19-11 

2.50 

3.5  -4.4 

11 

17.09 

2.43 

r  =  .64 


Table  XUl.  Relationship  Between  Mean  OER  and  Nominating 
Score  for  Officers  Screened  for  Promotion  to  Colonel 


MEAN  OER 

INTERVAL 

N 

NOMINATING  SCORE 

MEAN  SD 

8.5  -  9.0 

31 

26.45 

1.48 

7.5  -  8.4 

1325 

24.16 

2.23 

6.5  -  7.4 

1232 

21.83 

2.54 

5.5  -6.4 

205 

18.57 

2.57 

4.5  -  5.4 

20 

16.25 

1.76 

3.5 -4.4 

r  =  .65 

1 

15.00 

0.00 

Table  XIV. 

Relationship  Between  Mean  OER  and  Promotion 

Score  for  Officers  Considered  for  Promotion 

to  Colonel 

MEAN  OER 

PROMOTION  SCORE 

INTERVAL 

N 

MEAN 

SD 

8.5  -  9.0 

30 

26.17 

1.95 

7.5  -8.4 

906 

24.35 

2.07 

6.5  -  7.4 

270 

23.20 

2.02 

5.5  -  6.4 

2 

23.50 

1.50 

t  =  .36 

Relationships  between  Promotion  Scores  and  Mean  OERs  are  somewhat  lower  than  the 
reliabilities  of  the  Promotion  Scores.  This  might  be  interpreted  as  meaning  that  a  sort  of 
averaging  of  effectiveness  ratings  does  not  entirely  account  for  the  reliable  (or  consistent) 
part  of  Promotion  Scores.  It  may  be  that  the  different  OERs  are  given  differential  weight  in 
arriving  at  a  rating,  or  that  some  non-OER  data  are  considered  in  a  systematic  way  in  arriving 
at  a  promotion  rating. 

Data  reported  here  also  suggest  that  these  additional  considerations  (whether  differential 
weighting  of  OERs  or  noo*OER  factors)  are  probably  considered  systematically  for  all  eligible 
offi  '‘ts  regardless  of  average  level  of  performance  in  arriving  at  a  promotion  rating. 

VII.  CONCLUSIONS 

The  foregoing  analyses  have  indicated  that  Promotion  Board  evaluation  scores  and  the 
resulting  recommendations  regarding  promotion  are  reasonably  reliable  and  compare  favorably 
in  reliability  with  other  types  of  ratings.  Analysis  of  Promotion  Score  reliabilities  was  ap* 
proached  in  two  different  ways. 

First,  promotion  selections  based  on  colonels’  Nominating  Scores  were  compared  with 
the  selections  actually  made  later  by  the  Promotion  Board.  From  this  analysis,  it  was  found 
that  the  two  boards  would  have  agreed  on  their  promotion  decision  for  71  to  83  percent  of  ihe 
officers  considered.  There  was  some  evidence  that  the  Nominating  Board  may  have  perceived 
its  job  as  being  somewhat  different  from  that  of  the  later  Promotion  Board;  it  is  likely  that  the 
two  boards  would  have  agreed  on  a  higher  percentage  of  these  decisions  had  that  not  been  so. 

Secondly,  an  analysis  was  made  of  the  extent  to  which  panel  members  within  a  single 
board  agree  with  each  other,  and  these  analyses  were  used  as  a  basis  for  estimates  of  the 
extent  to  which  different  boards  with  the  same  frame  of  reference  would  agree  with  each  other. 
This  analysis  indicated  that  from  79  to  92  percent  of  the  promotion  decisions  made  by  the 
boards  would  have  been  concurred  in  by  a  hypothetical  second  board. 

Further  analysis  was  made  of  the  expected  boatd-to-board  disagreements  and  it  was 
found  that  they  are  concentrated  in  the  vicinity  of  the  selection  cutoff  score;  the  neater  to  the 
cutoff  score  a  ratee  is  placed  by  the  board  evaluation,  the  greater  the  likelihood  that  the  promo* 
tion  decision  about  him  would  be  reversed  by  a  second  board. 

It  is  further  suggested  that  data  other  than  an  average  of  OERs  ate  considered  by  Promo¬ 
tion  Boards  in  arriving  at  their  ratings,  and  that  these  as  yet  undefined  additional  considerations 
are  used  with  some  universality  by  raters. 

It  is  obvious  from  die  present  analyses  that  little  improvement  can  be  made  in  the  consist¬ 
ency  with  which  promotion  decisions  regarding  outstanding  or  extremely  poor  officers  are  made; 
if  Promotion  Board  procedures  are  improved  upon  with  regard  to  consistency  with  which  promo¬ 
tion  decisions  are  made,  efforts  at  improvement  must  be  concentrated  on  evaluation  of  "average” 
officers;  it  is  in  this  range  of  Air  Force  officer  performance  that  distinctions  in  the  respective 
merits  of  officers  are  more  difficult  to  make,  and  proximity  of  the  officer’s  evaluated  potential 
for  promotion  to  promotion  cutoff  increases  the  likelihood  that  chance  factors  may  influence 
his  selection  or  nonselection  for  promotion. 

Present  plans  call  for  analysis  of  relationships  of  certain  promotion  folder  data  to  Promo¬ 
tion  Scores  and  study  of  the  feasibility  of  use  of  predictions  based  on  these  relationships  as 
an  aid  in  monitoring  promotion  decisions  so  that  officers  about  whom  there  is  greatest  likeli¬ 
hood  of  an  "inconsistent”  or  unreliable  decision  may  be  identified  in  advance  and  brought  to 
the  attention  of  the  board  for  particularly  careful  evaluation. 


APPENDIX  I.  RELATIONSHIP  BETWEEN  NOMINATING  AND  PROMOTION  SCORES 

Table  XV  gives  the  correlations  between  Nominating  and  Promotion  Scores  for  officers 
considered  by  the  FY  62  Colonels  Promotion  Board.  Correlations  ate  given  for  each  combina¬ 
tion  of  Nominating  and  promotion  panels,  along  with  means  and  standard  deviations  for  the  two 
variables. 


Table  XV.  Cfithin-Panels  Coirelations  Between  Nominating 
Scores  (NS)  and  Promotion  Scores  (PS),  for  FY  62  Colonel  Boards 


PROMOTION  PANEL 

NOMINATING 

PANEL 

1 

2 

3 

A 

NS  PS 

NS  PS 

NS  PS 

NS  PS 

1 

f 

.64 

.35 

.33 

.67 

n 

38 

68 

38 

23 

M 

25.53  23.58 

25.16  24.53 

25.39  23.82 

25.22  22.87 

SD 

1.23  2.67 

1.12  1.91 

1.20  1.78 

1.25  1.54 

2 

f 

.72 

.23 

.55 

.58 

n 

17 

42 

60 

% 

M 

25.65  24.00 

24.79  24.69 

25.00  24.00 

25.25  24.39 

SD 

1.41  2.95 

.96  1.71 

1.00  1.90 

1.23  2.13 

3 

t 

.36 

.38 

.41 

.47 

n 

28 

44 

62 

33 

M 

23.82  22.93 

24.39  24.91 

23.98  23.97 

24.39  24.36 

SD 

1.10  2.17 

1.35  1.84 

1.13  1.75 

1.25  1.86 

4 

r 

.53 

.47 

.65 

.43 

n 

19 

46 

18 

68 

M 

25.47  24.47 

25.39  24.89 

25.06  24.11 

25.38  23.50 

SD 

1.09  2.62 

1.22  2.02 

1.03  1.49 

1.06  1.95 

5 

r 

.64 

.65 

.61 

.30 

n 

58 

53 

43 

21 

M 

25.2)  23.79 

25.38  24.68 

25.05  24.28 

25.33  23.76 

SD 

1.11  2.80 

1.29  2.01 

1.06  1.83 

1.21  2.04 

6 

r 

.46 

.50 

.50 

.35 

n 

40 

61 

59 

40 

M 

25.52  23.45 

25.66  24.54 

25.59  24.47 

25.68  23.60 

SD 

1.07  2.21 

1.11  1.91 

1.09  1.65 

1.08  2.06 

7 

r 

.46 

.63 

.73 

.63 

n 

39 

16 

56 

58 

M 

25.00  23.51 

25.25  25.56 

25.41  24.45 

25.17  24.31 

SD 

1.11  2.85 

1.20  1.54 

1.21  1.78 

1.16  2.27 
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APPEND'K  II.  METHODS  OF  DERIVING  RELlABlLlTi'  F^TIMATES 


Estimating  Reliability  of  Promotion  Scores 

Promotion  Scores  ate  obtained  by  adding  together  the  three  independent  ratings  made  by 
the  panel  members  after  examination  of  the  selection  folder.  Each  rating  can  range  from  $  to 
10  and  the  evaluation  scores  from  13  to  30. 

The  reliability  of  a  score  based  upon  the  sum  of  a  number  of  other  scores  can  be  estimated 
by  the  Kudet-Richardson  formula  20.  This  is  an  internal  consistency  reliability  estimate  and 
essentially  estimates  the  extent  to  which  the  score  would  correlate  with  another  score  obtained 
in  exactly  the  same  way.  In  the  present  case,  the  reliability  coefficient  is  an  estimate  of  the 
cortelation  between  the  evaluation  score  based  on  the  three  panel-member  ratings  and  another 
evaluation  score  ba«ed  on  three  more  panel-member  ratings. 

The  foimuta  used  in  estimating  the  reliabilities,  which  is  analogous  to  the  Kuder-Richardson 
formula  20,  is: 


where:  s  the  reliability  coefficient 

i  =  any  panel-member’s  rating 
s  -  evaluation  score 
n  =  number  of  members  per  panel  =  3 
^s  ^  =  variance  (squared  deviation)  of  the  evaluation  scores 
S^i  *  =  sum  of  the  variances  of  the  three  panel-member’s  ratings 

Since  cutting  scores  for  promotion  recommendations  are  established  separately  for  each 
panel,  reliability  estimates  were  computed  panel  by  panel  and  then  averaged  to  insure  that 
board  reliability  estimates  would  be  realistic.  To  cancel  out  any  possible  effect  of  practice 
or  boredom  on  the  reliability  estimates,  four  reliability  coefficients  (one  based  on  30  cases 
tated  early  in  the  cycle,  a  second  based  on30  rated  toward  the  middle,  the  third  based  on  30 
rated  after  the  middle,  and  the  fourth  based  on  30  rated  near  the  end)  were  computed  for  each 
panel  and  averaged  to  obtain  the  panel  reliability  estimates. 

This  resulted  in  reliability  estimates  which  were  based  on  a  total  of  approximately  840 
cases  for  the  Majors  Board,  700  cases  for  the  Lieutenant  Colonels  Board,  and  480  cases  for 
the  Colonels  Board. 

The  panel-by-panel  reliability  estimates,  along  with  the  panel  means  and  variances,  are 
presented  in  Tables  XVI,  XVU,  and  XVIIl.  These  tables  ate  of  interest,  since  they  provide 
indications  of  panel  differences  and  may  be  examined  for  trends  if  desired.  It  might  be  expected, 
for  example,  that  reliabilities  of  early  ratings  (because  of  lack  of  practice)  and  of  late  ratings 
(because  of  boredom)  would  be  lower  than  the  others.  No  such  tendencies  are  apparent. 

Constructing  Board-to-Board  Reliability  Tables 

As  noted  earlier,  die  reliability  estimates  may  be  regarded  as  estimates  of  the  correlation 
between  the  Promotion  Scores  of  a  Promotion  Board  and  the  Promotion  Scores  which  would  result 
were  another  equivalent  board  to  re-evaluate  the  same  eligible  officers.  Thus  the  reliability 
coefficient  can  be  used  as  a  correlation  coefficient  to  develop  a  regression  equation  to  predict 
the  Promotion  Scores  of  the  second  board  from  those  of  the  first  board.  The  mean  and  standard 
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Table  XVL  Panel  Means,  Variances,  and  Reliability 
Estimates  for  FY  62  Temporary  Majors  Board 
(N  =  30  cases  per  page) 


PANEL 

ROSTER 

PAGE 

M 

'll 

1 

1 

23.6 

2.2 

.744 

1 

42 

23.9 

5.2 

.940 

1 

114 

24.3 

6.2 

.908 

1 

167 

23.9 

9.5 

.940 

Average  .897 

2 

2 

24.9 

3.2 

.880 

2 

47 

25.1 

5.3 

.866 

2 

113 

24.4 

5.6 

.882 

2 

168 

24.6 

2.8 

.741 

Average  .851 

3 

3 

23.9 

3.4 

.G62 

3 

46 

24.0 

3.6 

.861 

3 

115 

23.2 

10.2 

.912 

3 

169 

22.7 

4.5 

.910 

Average  .889 

4 

4 

22.8 

5.7 

.914 

4 

44 

22.7 

3.3 

.740 

4 

111 

22.8 

5.9 

.816 

4 

166 

22.8 

7.6 

.904 

Average  .856 

5 

5 

22.8 

4.5 

.753 

5 

43 

22.6 

5.1 

.905 

5 

110 

22.9 

5.3 

.892 

5 

171 

22.5 

5.8 

.882 

Average  .869 

6 

6 

24.0 

8.2 

.940 

6 

41 

24.4 

6.8 

.882 

6 

109 

23.6 

7.1 

.860 

6 

173 

23.5 

4.6 

.837 

Average  .886 

7 

7 

21.6 

3.9 

.846 

7 

45 

22.3 

6.1 

.927 

7 

112 

23.2 

8.2 

.951 

7 

172 

24.2 

3.1 

.876 

Average  .907 
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Table  XVII.  Panel  Means,  Variances,  and  Reliability 
Estimates  for  FY  62  Temporary  Lt  Colonels  Board 

(N  =  30  cases  per  page) 


ROSTER 

PANEL 

PACE 

M 

'11 

1 

1 

23.6 

4.9 

.873 

1 

72 

25.3 

5.1 

.870 

1 

151 

23.2 

7.9 

.938 

1 

239 

23.2 

8.0 

.942 

Average  .906 

2 

2 

21.3 

3.8 

.807 

2 

70 

23.4 

7.8 

.880 

2 

148 

24.0 

6.4 

.914 

2 

222 

23.8 

4.4 

.897 

Average  .875 

3 

3 

23.6 

5.6 

.83! 

3 

223 

22.2 

5.4 

.879 

Average  .855 

4 

4 

25.6 

7.6 

.904 

4 

74 

23.7 

9.3 

.872 

4 

149 

24.5 

6.5 

.902 

4 

221 

24.0 

8.4 

.932 

Average  .902 

5 

5 

23.6 

5.4 

.890 

5 

68 

23.8 

7.7 

.876 

5 

146 

23.5 

11.2 

.939 

5 

220 

25.3 

4.5 

.840 

Average  .866 

6 

6 

25.0 

5.6 

.900 

6 

75 

23.9 

4.4 

.862 

6 

152 

23.2 

7.9 

.938 

6 

217 

23.4 

13.7 

.954 

Average  .914 


7 

7 

23.5 

6.8 

.894 

7 

73 

24.8 

6.6 

.910 

7 

150 

23.8 

6.1 

.912 

7 

218 

21.8 

7.5 

910 

Average  .904 
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Table  XVIll.  Panel  Means,  Variances,  and  Reliability 
Estimates  for  FY  62  Temporary  Colonels  Board 


(N  =  30  cases  per  puf^e) 


PANEL 

ROSTER  P.'.GE 

MEAN 

(7* 

1 

1 

23.4 

7.4 

.806 

1 

14 

23.4 

6.4 

.834 

1 

25 

24.7 

7.8 

.881 

1 

34 

23.0 

6.3 

.756 

Average  .825 

2 

2 

25.3 

4.7 

.837 

2 

17 

24.7 

3.4 

.763 

y 

26 

24.1 

2.6 

.692 

2 

40 

24.1 

5.0 

.840 

Average  .790 

3 

3 

23.8 

4.0 

.824 

3 

15 

24.2 

2.3 

.64S 

3 

28 

23.9 

3.2 

.631 

3 

37 

25.3 

3.2 

.753 

Average  .725 

4 

4 

24.4 

4.6 

.741 

4 

16 

23.8 

5.1 

.760 

4 

27 

24.2 

5.3 

.806 

4 

38 

24.8 

3.8 

.678 

Average  .749 

deviation  of  the  second  board  scores  are  assumed  to  be  the  same  as  the  mean  and  standard 
deviation  of  the  first  board  scores.  Such  an  equation  was  developed  for  each  of  the  three 
Promotion  Boards,  resulting  in  the  following: 

FY  62  Majors:  Board  2  Score  =  .88  times  Board  1  Score  +  2.84 

FY  62  Lt  Cols:  Board  2  Score  =  .90  times  Board  1  Score  4  2.39 

FY  62  Colonels;  Board  2  Score  =  .77  times  Board  1  Score  +  5.34 

Since  the  correlations  between  Board  1  and  Board  2  ate  less  than  1.00,  for  each  Board 
1  Score,  there  will  be  a  distribution  of  Board  2  scores.  This  distribution  will  have  a  mean 
equal  to  the  predicted  Board  2  Score  and  a  standard  deviation  equal  to  the  standard  error  of 
estimating  Board  2  from  Board  1.  These  standard  errors  of  estimate  are  respectively,  1.14, 
1.18,  and  1.34,  for  the  Majors,  Lieutenant  Colonels,  and  Colonels  Boards. 

To  develop  the  reliability  tables  (Tables  2,  3,  and  4),  for  each  Board  1  (Promotion  Board) 
score,  the  corresponding  Board  2  score  was  predicted.  The  differences  between  this  predicted 
score  and  the  various  possible  whole  number  Board  2  scores  were  then  obtained  by  subtraction. 
(Since  a  score  of  27  is  assumed  to  covet  the  range  between  26.5  and  27.5,  the  26.3  was  used  in 
obtaining  the  difference  and  similarly  for  other  scores.)  Each  difference  was  then  divided  by 
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the  standard  error  of  estimate  and  the  results  used  to  ..  nter  the  normal  probability  table  to  obta..i 
the  proportion  of  cases  expected  to  have  a  difference  sore  of  that  magnitude.  The  number  of 
cases  having  each  Board  1  Score  was  then  multiplied  by  the  appropriate  proportions  to  obtain 
the  expected  distribution  of  Board  2  Scores.  In  this  way  the  various  columns  of  the  table  were 
developed.  It  should  of  course  be  noted  that  the  column  values  are  expected  values  and  thus 
only  estimates  of  the  column  distributions  that  would  be  obtained  were  another  board  actually 
to  re-evaluate  the  eligibles. 
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